Electronic device for performing object detection and operation method thereof

ABSTRACT

An electronic device includes: a first image sensor that outputs a first image produced by photographing a first viewing angle; a second image sensor that outputs a second image produced by photographing a second viewing angle that overlaps a portion of the first viewing angle; a third image sensor that outputs a third image produced by photographing a third viewing angle; and a processor that performs object detection on an object included in an image. The processor generates disparity information indicating a separation degree of a feature point of the first and second images, transforms the third image based on the disparity information, and performs object detection on the transformed third image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No.10-2020-0064597, filed on May 28, 2020, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND

The disclosure relates to an electronic device for performing objectdetection and an operation method thereof, and more particularly, to anelectronic device for performing object detection by using image sensorshaving photographing areas, which overlap each other, and an operationmethod of the electronic device.

A self-driving system (or an Advanced Driver Assistance System (ADAS))may obtain information regarding a host vehicle and a surroundingenvironment from various types of sensors and may safely navigate bycontrolling the host vehicle based on the obtained information. Indetail, the self-driving system may capture images of a surroundingenvironment of the host vehicle by using image sensors, perform objectdetection on the captured images, and may control a driving direction,speed, and the like of the host vehicle according to an object detectionresult.

The self-driving system may include an image sensor that mainlyphotographs a front view of the host vehicle and perform objectdetection on an image of the front view. When an object comes close to aleft side or a right side of the host vehicle, the image of the frontview may include only part of the object. Accordingly, it is difficultfor the self-driving system to accurately detect, from the image of thefront view, the object coming close to the left side or the right sideof the host vehicle.

SUMMARY

According to one or more embodiments, an electronic device detects animage area corresponding to a proximity object based on two imagescaptured in one direction, merges an image, which is captured in anotherdirection and includes the proximity object, with the detected imagearea, and performs object detection on a merged image.

An electronic device includes a first image sensor configured to outputa first image produced by photographing a first photographing area. Asecond image sensor outputs a second image produced by photographing asecond photographing area that overlaps at least some portions of thefirst photographing area. A third image sensor outputs a third imageproduced by photographing a third photographing area. A processorperforms object detection on at least one object included in an image.The processor generates disparity information indicating a separationdegree of at least one feature point of the first image and the secondimage, transforms the third image based on the disparity information,and performs the object detection on the transformed third image.

According to one or more embodiments, an electronic device includes afirst image sensor configured to output a first color image captured ina first direction. A depth sensor outputs a depth image corresponding tothe first color image. A second image sensor outputs a second colorimage captured in a second direction. A processor performs objectdetection on at least one object included in an image. The processortransforms the second color image based on the first color image and thedepth image and performs the object detection on the second color imagethat is transformed.

According to one or more embodiments, an operation method of anelectronic device includes obtaining a first image produced byphotographing a first photographing area; obtaining a second imageproduced by photographing a second photographing area that overlaps atleast some portions of the first photographing area; obtaining a thirdimage produced by photographing a third photographing area; generatingdisparity information indicating a separation degree of at least onefeature point of the first image and the second image; transforming thethird image based on the disparity information; and performing objectdetection on the third image that is transformed.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the disclosure will be more clearly understood from thefollowing detailed description taken in conjunction with theaccompanying drawings in which:

FIG. 1 is a block diagram of an electronic device according to anembodiment;

FIG. 2 is a diagram illustrating photographing areas of image sensors,according to an embodiment;

FIG. 3 is a diagram of images captured by image sensors, according to anembodiment;

FIG. 4 is a flowchart of an operation method of an electronic device,according to an embodiment;

FIG. 5 is a flowchart of a method of generating disparity information,according to an embodiment;

FIGS. 6A and 6B are diagrams of a method of generating disparityinformation, according to an embodiment;

FIG. 7 is a flowchart of an image transformation method, according to anembodiment;

FIGS. 8A and 8B are diagrams of the image transformation method of FIG.7 ;

FIG. 9 is a flowchart of an image transformation method according to anembodiment;

FIGS. 10A and 10B are diagrams of the image transformation method ofFIG. 9 ;

FIG. 11 is a block diagram of an image transformation module accordingto an embodiment;

FIG. 12 is a block diagram of an electronic device according to anembodiment;

FIG. 13 is a flowchart of an operation method of an electronic device,according to an embodiment;

FIG. 14 is a diagram of a host vehicle including an electronic device,according to an embodiment; and

FIG. 15 is a block diagram of a self-driving device according to anembodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 is a block diagram of an electronic device 10 according to anembodiment.

Referring to FIG. 1 , the electronic device 10 may include a sensor 100,a memory 200, and a processor 300. The electronic device 100 may be adevice that performs image processing on images. The image processingmay be a process of analyzing an image and performing object detectionor segmentation on at least one object included in the image. Types ofthe image processing are not limited thereto and may include variousprocesses.

The electronic device 10 may be realized as a personal computer (PC), anInternet of Things (IoT) device, or a portable electronic device. Theportable electronic device may be included in various devices such as alaptop computer, a mobile phone, a smart phone, a tablet PC, a personaldigital assistant (PDA), an enterprise digital assistant (EDA), adigital still camera, a digital video camera, an audio device, aportable multimedia player (PMP), a personal navigation device (PND), anMP3 player, a handheld game console, an e-book reader, and a wearabledevice.

In an embodiment, the electronic device 10 may be a device that controlsa host vehicle. The electronic device 10 may perform object detectionbased on images capturing a surrounding environment of the host vehicleand control the host vehicle according to an object detection result.Hereinafter, for convenience, it is assumed that the electronic device10 is a device that controls the host vehicle.

The sensor 100 may include sensors that generate information regardingthe surrounding environment. For example, the sensor 100 may include animage sensor such as a Charge Coupled Device (CCD) or a ComplementaryMetal Oxide Semiconductor (CMOS). In an example, the sensor 100 mayinclude a first image sensor 110, a second image sensor 120, and a thirdimage sensor 130.

The first image sensor 110 may output a first image IMG1 of a firstphotographing area, the second image sensor 120 may output a secondimage IMG2 of a second photographing area, and the third image sensor130 may output a third image IMG3 of a third photographing area. In anembodiment, the first image sensor 110 and the second image sensor 120may be arranged adjacent to each other and capture images in the samedirection or a similar direction. Accordingly, the first photographingarea of the first image sensor 110 may overlap most of the secondphotographing area of the second image sensor 120. As a gap between thefirst image sensor 110 and the second image sensor 120 is small, aregion where the first photographing area overlaps the secondphotographing area may increase. The first image sensor 110 and thesecond image sensor 120 may each be realized as a stereo camera (notshown). Also, a first image IMG1 and a second image IMG2 may be referredto as stereo images.

In an embodiment, the third image sensor 130 may capture an image in adirection perpendicular to a photographing direction of the first imagesensor 110 or the second image sensor 120. For example, the first imagesensor 110 and the second image sensor 120 may photograph a front viewof the host vehicle, and the third image sensor 130 may photograph aside view of the host vehicle. As another example, the first imagesensor 110 and the second image sensor 120 may photograph a rear view ofthe host vehicle, and the third image sensor 130 may photograph the sideview of the host vehicle.

In the above examples, according to embodiments, the third image sensor130, which photographs the side view of the host vehicle, may include atleast two image sensors photographing a left side view and/or a rightside view of the host vehicle. For example, when the third image sensor130 includes two image sensors and photographs one of the left side viewand the right side view of the host vehicle, at least two image sensorsmay have photographing areas overlapping each other. As another example,when the third image sensor 130 includes two image sensors andphotographs the left and right side views of the host vehicle, at leasttwo image sensors may include different photographing areas.

For convenience of explanation, hereinafter, it is assumed that thefirst image sensor 110 and the second image sensor 120 photograph thefront view of the host vehicle and that the third image sensor 130includes one image sensor and photographs the side view of the hostvehicle.

The third photographing area of the third image sensor 130 may overlapat least one of the first photographing area and the secondphotographing area. Because the third image sensor 130 captures an imagein a vertical direction, overlapping of the third photographing areawith the first or second photographing area may be relatively smallerthan overlapping of the first photographing area with the secondphotographing area. A photographing direction of the third image sensor130 is not limited thereto. The photographing direction of the thirdimage sensor 130 is a direction in which the third photographing areaoverlaps the first or second photographing area.

When the object (e.g., a peripheral vehicle) comes close to a left sideor a right side of the front of the electronic device 10, only part ofthe object (e.g., a front portion of the peripheral vehicle) may beincluded in the first image IMG1 or the second image IMG2. Also, whenthe object is located in a photographing direction of the third imagesensor 130, other portions of the object (e.g., middle and rear portionsof the peripheral vehicle) may be included in the third image IMG3. Inthis case, although the first image IMG1, the second image IMG2, and thethird image IMG3 are analyzed, it may be difficult for the processor 300to detect a proximity object close to the electronic device 10.

As a storage in which data is stored, the memory 200 may store datagenerated by the sensor 100 and various pieces of data generated whilethe processor 300 performs calculations. For example, the memory 200 maystore the first to third images IMG1 to IMG3 that are obtained by thefirst to third image sensors 110 to 130. As described below with regardto the operation of the processor 300, the memory 200 may store aprocessing result according to the image processing of the processor300.

The processor 300 may control all operations of the electronic device10. The processor 300 may include various operation processors such as aCentral Processing Unit (CPU), a Graphic Processing Unit (GPU), anApplication Processor (AP), a Digital Signal Processor (DSP), aField-Programmable Gate Array (FPGA), a Neural network Processing unit(NPU), an Electronic Control Unit (ECU), and an Image Signal Processor(ISP).

The processor 300 according to an embodiment may include an imagetransformation module 310. The image transformation module 310 maytransform the third image IMG3 based on the first and second images IMG1and IMG2. In an embodiment, the image transformation module 310 maygenerate disparity information indicating a separation degree of atleast one common feature point of the first image IMG1 and the secondimage IMG2 and may transform the third image IMG3 based on the generateddisparity information. An image transformation operation of the imagetransformation module 310 will be described below in more detail.

Because the first image sensor 110 and the second image sensor 120 arearranged adjacent to each other, at least one object may be commonlyincluded in the first image IMG1 and the second image IMG2. Therefore,feature points of the object that is commonly included may be commonlyincluded in the first image IMG1 and the second image IMG2 as well.Here, the term ‘feature point’ is a point indicating a feature of anobject and denotes pixels forming the object. The image transformationmodule 310 may detect a common feature point by analyzing the firstimage IMG1 and the second image IMG2 and may generate the disparityinformation based on a difference between a location of the featurepoint of the first image IMG1 and a location of the feature point of thesecond image IMG2.

The disparity information may have different values according to adistance between the electronic device 10 and the object. For example,when the first object is in a remote place, the difference between thelocation of the feature point of the first image IMG1 and the locationof the feature point of the second image IMG2 is small. Therefore, adisparity value of a feature point of a first object may be relativelysmall. As another example, when a second object comes close to theelectronic device 10, the difference between the location of the featurepoint of the first image IMG1 and the location of the feature point ofthe second image IMG2 is great. Therefore, a disparity value regardingthe feature point of the second object may be relatively great. Theimage transformation module 310 may transform the third image IMG3 byusing features of the above disparity values.

In detail, the image transformation module 310 may detect an area, whichcorresponds to the proximity object close to the electronic device 10,from the first image IMG1 or the second image IMG2, based on thedisparity information. For example, because a disparity value regardinga feature point forming the proximity object is relatively great, theimage transformation module 310 may detect an area of the first imageIMG1 or the second image IMG2, which has a great disparity value, as anarea corresponding to the proximity object.

The image transformation module 310 may merge an area including thedetected area with the third image IMG3. For example, the imagetransformation module 310 may merge an area of the first image IMG1 orthe second image IMG2, which corresponds to a portion (e.g., a frontportion of the peripheral vehicle) of the proximity object, with an areaof the third image IMG3, which relates to other portions (e.g., middleand rear portions of the peripheral vehicle) of the proximity object,thereby transforming the third image IMG3 to include the entire object.In the above examples, it is described that a portion of the proximityobject is included in the first image IMG1 or the second image IM2 andthat other portions of the proximity object are included in the thirdimage IMG3, but this is merely an example. Portions of the proximityobject may be overlappingly included in the first to third images IMG1to IMG3.

The processor 300 may include an object detection module 320. The objectdetection module 320 may detect at least one object included in animage. In an embodiment, the object detection module 320 may detect anobject included in at least one of the first to third images IMG1 toIMG3. Also, the object detection module 320 may detect an objectincluded in the third image IMG3 that is converted by the imagetransformation module 310. In detail, the object detection module 320may detect the object from the third image IMG3 transformed to includethe entire proximity object that is close to the electronic device 10.

The image transformation module 310 and the object detection module 320may each be realized as firmware or software and may be loaded on thememory 200 and executed by the processor 300. However, one or moreembodiments are not limited thereto. The image transformation module 310and the object detection module 320 may each be realized as hardware ora combination of software and hardware.

FIG. 1 illustrates that the electronic device 10 includes the first tothird image sensors 110 to 130. However, according to embodiments, theelectronic device 10 may not include the first to third image sensors110 to 130 or may include at least some of them and may receive at leastone of the first to third images IMG1 to IMG3 from an external device.

Also, FIG. 1 illustrates that the electronic device 10 includes thememory 200, but according to an embodiment, the electronic device 10 maybe separated from the memory 200.

The electronic device 10 according to an embodiment may detect an imagearea corresponding to the proximity object by using two images capturedin one direction, merge an image, which is captured in another directionand includes the proximity object, with the detected area, and performthe object detection on the merged image, thereby accurately detectingthe proximity object.

Hereinafter, referring to FIGS. 2 and 3 , features of the first to thirdimages IMG1 to IMG3 captured by the first to third image sensors 110 to130 of the electronic device 10 will be described.

FIG. 2 is a diagram illustrating photographing areas of the first tothird image sensors 110 to 130, according to an embodiment. In detail,FIG. 2 illustrates the photographing areas of the first image sensor110, the second image sensor 120, and the third image sensor 130 of FIG.1 .

Referring to FIGS. 1 and 2 , the electronic device 10 may be on the hostvehicle and photograph a surrounding environment of the host vehicle.The first image sensor 110 may photograph a front view (left) of thehost vehicle, the second image sensor 120 may photograph a front view(right) of the host vehicle, and the third image sensor 130 mayphotograph a side view of the host vehicle. FIG. 2 illustrates that thethird image sensor 130 photographs a right side view of the hostvehicle, but this is merely an example. The third image sensor 130 mayphotograph a left side view of the host vehicle. As described, theelectronic device 10 may capture images of situations around the hostvehicle by using the first to third image sensors 110 to 130 and mayperform object detection on the captured images.

Objects OB1 to OB4 may be in vicinity of the electronic device 10.Referring to FIG. 2 , the first object OB1, the second object OB2, andthe fourth object OB4 may be located in respective photographing areasof the first image sensor 110 and the second image sensor 120. The thirdobject OB3 may only be partially present in the photographing area ofeach of the first to third image sensors 110 to 130.

FIG. 3 is a diagram of images captured by image sensors, according to anembodiment. In detail, FIG. 3 is a diagram illustrating first to thirdimages IMG1 to IMG3 captured by the first to third images sensors 110 to130 of FIG. 1 and including the first to fourth objects OB1 to OB4 ofFIG. 2 .

Referring to FIGS. 1 to 3 , the first image IMG1 captured by the firstimage sensor 110 and the second image IMG2 captured by the second imagesensor 120 may include the first object OB1, the second object OB2, andthe fourth object OB4. Therefore, the electronic device 10 may detectthe first object OB1, the second object OB2, and the fourth object OB4by performing the object detection on the first image IMG1 or the secondimage IMG2.

The first image IMG1 and the second image IMG2 may include only a frontportion of the third object OB3 that is the closest to the host vehicle.The third image IMG3 captured by the third image sensor 130 may includea middle portion and a rear portion of the third object OB3. Therefore,the electronic device 10 may unlikely detect the third object OB3 eventhough the object detection is performed on the first image IMG1, thesecond image IMG2, or the third image IMG3.

The processor 300 may transform the third image IMG3 to accuratelydetect the third object OB3. In detail, the processor 300 may generatedisparity information of the first image IMG1 and the second image IMG2.Also, the processor 300 may detect an area of the first image IMG1 orthe second image IMG2 (e.g., a front portion), which corresponds to thethird object OB3, based on the generated disparity information. Theprocessor 300 may transform the third image IMG3 by merging the detectedarea (e.g., the front portion) with an area associated with the thirdobject OB3 of the third image IMG3. The processor 300 may detect thethird object OB3 by performing the object detection on the transformedthird image IMG3.

FIG. 4 is a flowchart of an operation method of an electronic device 10,according to an embodiment. In detail, FIG. 4 is a flowchart of anoperation method of the electronic device 10 of FIG. 1 . At least someof operations of FIG. 4 may be performed by the processor 300 by usingthe image transformation module 310 and the object detection module 320.

Referring to FIGS. 1 and 4 , in operation S110, the electronic device 10may obtain the first image IMG1 of the first photographing area. Indetail, the electronic device 10 may obtain the first image IMG1 of thefirst photographing area that is captured by the first image sensor 110.However, one or more embodiments are not limited thereto, and theelectronic device 10 may obtain the first image IMG1 from an externaldevice.

In operation S120, the electronic device 10 may obtain the second imageIMG2 of the second photographing area that overlaps at least someportions of the first photographing area. In detail, the electronicdevice 10 may obtain the second image IMG2 of the second photographingarea that is captured by the second image sensor 120. However, one ormore embodiments are not limited thereto, and the electronic device 10may obtain the second image IMG2 from the external device.

In operation S130, the electronic device 10 may obtain the third imageIMG3 of the third photographing area. In detail, the electronic device10 may obtain the third image IMG3 of the third photographing area thatis captured by the third image sensor 130. However, one or moreembodiments are not limited thereto, and the electronic device 10 mayobtain the third image IMG3 from the external device. The thirdphotographing area may overlap the first photographing area or thesecond photographing area.

In operation S140, the electronic device 10 may generate the disparityinformation. In detail, the electronic device 10 may generate thedisparity information indicating a separation degree of at least onecommon feature point of the first image IMG1 and the second image IMG2.

In operation S150, the electronic device 10 may transform the thirdimage IMG3 based on the generated disparity information. In detail, theelectronic device 10 may detect the area, which indicates the proximityobject close to the electronic device 10, from the first image IMG1 orthe second image IM2 based on the disparity information and maytransform the third image IMG3 by merging the detected area with thethird image IMG3. In operation S160, the electronic device 10 mayperform the object detection on the transformed third image IMG3.

FIG. 5 is a flowchart of a method of generating the disparityinformation, according to an embodiment. In detail, FIG. 5 is a diagramfor explaining a detailed operation (operation S140) of the method ofgenerating the disparity information of FIG. 4 . At least some ofoperations of FIG. 5 may be performed by the processor 300 by using theimage transformation module 310.

Referring to FIGS. 1, 4 and 5 , in operation S141, the electronic device10 may detect at least one first feature point included in the firstimage IMG1. In operation S143, the electronic device 10 may detect atleast one second feature point included in the second image IMG2. Thefirst image IMG1 and the second image IMG2 are respectively captured bythe first image sensor 110 and the second image sensor 120 that have thephotographing areas that overlap each other, and thus, the first imageIMG1 and the second image IMG2 may commonly include at least one object.

In operation S145, the electronic device 10 may perform feature matchingbetween the first feature point and the second feature point. In detail,the electronic device 10 may match the first and second feature points,which correspond to each other, for each of objects that are commonlyincluded in the first image IMG1 and the second image IMG2.

In operation S147, the electronic device 10 may calculate a separationdegree of the matched feature points. In detail, the electronic device10 may generate the disparity information by calculating a differencebetween the locations of the first feature point and the second featurepoint that are matched. A method whereby the electronic device 10generates the disparity information is not limited thereto, and thedisparity information may be generated in various manners.

FIGS. 6A and 6B are diagrams of a method of generating the disparityinformation, according to an embodiment.

Referring to FIGS. 1 and 6A, the electronic device 10 may detect atleast one first feature point included in the first image IMG1. Forexample, in operation S141, the electronic device 10 may detect at leastone first feature point constituting an object, for example, aneighboring vehicle, a road, a tree, or the like, which is included inthe first image IMG1. In operation S143, the electronic device 10 maydetect at least one second feature point constituting an object, forexample, a neighboring vehicle, a road, a tree, or the like, which isincluded in the second image IMG2.

In operation S145, the electronic device 10 may match the first andsecond feature points that are detected. For example, for a peripheralvehicle that is commonly included in the first image IMG1 and the secondimage IMG2, the electronic device 10 may match a first feature pointconstituting the peripheral vehicle in the first image IMG1 and a secondfeature point constituting the peripheral vehicle in the second imageIMG2. The electronic device 10 may perform feature matching identicallyon roads, trees, and the like that are commonly included in the firstimage IMG1 and the second image IMG2.

The electronic device 10 may determine areas of the first image IMG1 andthe second image IMG2, which include the matched feature points, andgenerate the disparity information by using the determined areas. Forexample, because the first image IMG1 is an image captured by the firstimage sensor 110 located on a left side of the second image sensor 120,a left edge of the first image IMG1 may correspond to an area excludedfrom the photographing area of the second image sensor 120. Therefore,the electronic device 10 may not detect the second feature point that ismatched with the first feature point that is located on the left edge ofthe first image IMG1. Accordingly, the electronic device 10 maydetermine that a region A except for the left edge of the first imageIMG1 is used to generate the disparity information.

Also, because the second image IMG2 is an image captured by the secondimage sensor 120 located on a right side of the first image sensor 110,a right edge of the second image IMG2 may be an area excluded from thephotographing area of the first image sensor 110. Therefore, theelectronic device 10 may not detect the first feature point matched withthe second feature point that is located on the right edge of the secondimage IMG2. Accordingly, the electronic device 10 may determine that aregion B except for the right edge of the second image IMG2 is used togenerate the disparity information.

Referring to FIGS. 1 and 6B, in operation S147, the electronic device 10may calculate disparity values that indicate the separation degree ofthe feature points by using the Region A and the Region B. For example,the electronic device 10 may calculate the separation degree bydeducting a location value of the second feature point of the secondimage IMG2 from a location value of the first feature point of the firstimage IMG1. Because the left edge of the first image IMG1 does notinclude the second feature point that is matched, the left edge may beset to have a value that is set in advance. Also, because the right edgeof the second image IMG2 does not include the first feature point thatis matched, the right edge may not be used to calculate the separationdegree. The electronic device 10 may generate disparity informationInfo_D by merging disparity values indicating the separation degree ofeach feature point.

FIG. 7 is a flowchart of an image transformation method, according to anembodiment. In detail, FIG. 7 is a diagram for explaining a detailedoperation (S150) of the image transformation method, based on thedisparity information of FIG. 4 . At least some of operations of FIG. 7may be performed by the processor 300 by using the image transformationmodule 310.

Referring to FIGS. 1, 4, and 7 , in operation S151, the electronicdevice 10 may extract a target area from the first image IMG1 or thesecond image IMG2. Here, the target area may be an area of the firstimage IMG1 or the second image IMG2 to be merged with the third imageIMG3. In an embodiment, the electronic device 10 may extract the targetarea from the first image IMG1 or the second image IMG2, based on thedisparity information. For example, the electronic device 10 mayextract, as a target area, an area of the first image IMG1 or the secondimage IMG2 that has a disparity value that is equal to or greater than athreshold value. Here, the threshold value may be a disparity value thatthe electronic device 10 may have when the object comes close theretoand may be set by a manufacturer or a user.

In another embodiment, the electronic device 10 may extract an area,which overlaps the third photographing area of the third image sensor130, from the first image IMG1 as the target area. Alternatively, theelectronic device 10 may extract an area, which overlaps the thirdphotographing area of the third image sensor 130, from the second imageIMG2 as the target area.

In operation S153, the electronic device 10 may transform the thirdimage IMG3 by merging the extracted target area with the third imageIMG3. In detail, the electronic device 10 may warp the target area andmay merge the warped target area with the third image IMG3.

In an embodiment, the electronic device 10 may include mappinginformation including a coordinate value of the third image IMG3 thatcorresponds to each coordinate value of the first image IMG1 or thesecond image IMG2. The electronic device 10 may identify thecorresponding coordinate value of the third image IMG3 for each pixelforming the target area, based on the mapping information. Theelectronic device 10 may merge a pixel value of each pixel forming thetarget area with the identified coordinate value of the third imageIMG3.

In another embodiment, for quicker calculation, the electronic device 10may only detect a coordinate value of the third image IMG3 correspondingto each feature point included in the target area, instead of each pixelforming the target area. The electronic device 10 may merge apreset-sized image (that is, a portion of the target area), whichincludes each feature point, with the detected coordinate value of thethird image IMG3.

When merging the target area with the third image IMG3, the electronicdevice 10 may merge, with the third image IMG3, the pixel values of thetarget area as well as the disparity values corresponding to the targetarea. For example, the electronic device 10 may merge the disparityvalues corresponding to the target area with the third image IMG3. Asanother example, the electronic device 10 may generate depth informationindicating a depth value based on the disparity values corresponding tothe target area and may merge the generated depth information with thethird image IMG3. As another example, the electronic device 10 maygenerate distance information indicating a distance to the host vehiclebased on the generated depth information and may merge the generateddistance information with the third image IMG3. A method whereby theelectronic device 10 merges the disparity values with the third imageIMG3 is not limited thereto. The electronic device 10 may perform theobject detection based on the disparity values and the pixel values ofthe third image IMG3 that is transformed.

A method whereby the electronic device 10 merges the target area withthe third image IMG3 is not limited thereto and may vary.

FIGS. 8A and 8B are diagrams of the image transformation method of FIG.7 .

Referring to FIGS. 1, 7 and 8A, the electronic device 10 may detecttarget disparity values Info_D(Target), which are equal to or greaterthan the threshold value, from the disparity information Info_D. Inoperation S151, the electronic device 10 may extract the target areabased on the detected target disparity values Info_D(Target) and thefirst image IMG1. In detail, the electronic device 10 may extract anarea of the first image IMG1, which has the detected target disparityvalues Info_D(Target), as the target area IMG_TG.

Referring to FIGS. 1, 7, and 8B, the electronic device 10 may generate atransformed third image IMG3_T by merging an extracted target areaIMG_TG with the third image IMG3 in operation S153. For example, theelectronic device 10 may detect the coordinate value of the third imageIMG3 that corresponds to each feature point included in the target areaIMG_TG. The electronic device 10 may merge an image, which has a presetsize and includes each feature point, with the identified coordinatevalue of the third image IMG3. In this case, according to an embodiment,the electronic device 10 may merge, with the detected coordinate valueof the third image IMG3, the disparity values corresponding to thepreset-sized image including each feature point. A method of merging thedisparity values with the third image IMG3 may be substantially the sameas the above-described method of FIG. 7 , and thus, descriptions of themethod will be omitted.

The coordinate value of the third image IMG3 corresponding to thefeature point may exceed a coordinate value range of pixels forming theexisting third image IMG3. Therefore, the transformed third image IMG3_Tmay have a greater size than the existing third image IMG3.

FIGS. 8A and 8B illustrate that the first image IMG1 is used, but one ormore embodiments are not limited thereto. The second image IMG2 may beused.

FIG. 9 is a flowchart of an image transformation method according to anembodiment. In detail, FIG. 9 is a diagram of a modifiable embodiment ofFIG. 7 . That is, FIG. 9 is a diagram for explaining a detailedoperation (S150) of the image transformation method based on thedisparity information of FIG. 4 . At least some of operations of FIG. 9may be performed by the processor 300 of the electronic device 10.

Referring to FIGS. 1, 4, and 9 , in operation S155, the electronicdevice 10 may mask areas of the first image IMG1 or the second imageIMG2 other than the target area instead of extracting the target areafrom the first image IMG1 or the second image IMG2.

In operation S157, the electronic device 10 may merge the masked imagewith the third image IMG3. A method of merging the masked image with thethird image IMG3 may be substantially the same as the above-describedmethod of FIG. 7 , and thus, descriptions thereof will be omitted.

FIGS. 10A and 10B are diagrams of the image transformation method ofFIG. 9 .

Referring to FIGS. 1, 9 and 10A, the electronic device 10 may detect thetarget disparity values Info_D(Target), which are equal to or greaterthan the threshold value, from the disparity information Info_D. Theelectronic device 10 may generate a masked first image IMG1_M by maskingthe first image IMG1 based on the first image IMG1 and the detectedtarget disparity values Info_D(Target) in operation S155. In detail, theelectronic device 10 may detect the target area corresponding to thetarget disparity values Info_D(Target) from the first image IMG1 and maymask other areas of the first image IMG1 than the target area.

Referring to FIGS. 1, 9, and 10B, the electronic device 10 may generatethe transformed third image IMG3_T by merging the masked first imageIMG1_M with the third image IMG3 in operation S157. In this case,according to an embodiment, the electronic device 10 may merge thedisparity values corresponding to the masked first image IMG1_M with thethird image IMG3. A method of merging the disparity values with thethird image IMG3 may be substantially the same as the above-describedmethod of FIG. 7 , and thus, descriptions thereof will be omitted. Theelectronic device 10 may not perform the object detection on the maskedarea when the object detection is performed on the transformed thirdimage IMG3_T.

Referring to FIGS. 10A and 10B, the first image IMG1 is used, but one ormore embodiments are not limited thereto. The second image IMG2 may beused.

FIG. 11 is a block diagram of the image transformation module 310according to an embodiment. In detail, FIG. 11 is a diagram of amodifiable embodiment of the image transformation module 310 of FIG. 1 .

Referring to FIG. 11 , the image transformation module 310 may include afirst artificial intelligence (AI) model 311 and a second AI model 313.The first AI model 311 may be an AI model that is trained to receiveimages and generate disparity information regarding the images based onthe received images. The second AI model 313 may be an AI model that istrained to receive disparity information and images and generate imagestransformed based on the received disparity information and images.

In an embodiment, the first AI model 311 may receive the first imageIMG1 and the second image IMG2 and generate the disparity informationInfo_D based on the received first and second images IMG1 and IMG2. Forexample, the first AI model 311 may receive the first image IMG1 and thesecond image IMG2 from the first image sensor 110 (of FIG. 1 ) and thesecond image sensor 120 (of FIG. 1 ) and may generate the disparityinformation Info_D regarding the received first and second images IMG1and IMG2.

The second AI model 313 may receive the disparity information Info_D andthe first to third images IMG1 to IMG3 and may transform the third imageIMG3 based on the received disparity information Info_D and the receivedfirst and second images IMG1 and IMG2. For example, the second AI model313 may receive the first to third images IMG1 to IMG3 from the first tothird image sensors 110 to 130 (of FIG. 1 ), receive the disparityinformation Info_D from the first AI model 311, transform the thirdimage IMG3 based on the first and second images IMG1 and IMG2, andoutput the transformed third image IMG3_T.

The second AI model 313 may generate the transformed third image IMG3_Tby merging, with the third image IMG3, pixel values of some areas of atleast one of the first image IMG1 and the second image IMG2.Alternatively, according to an embodiment, the second AI model 313 maygenerate the transformed third image IMG3_T by merging, with the thirdimage IMG3, the pixel values of areas of at least one of the first imageIMG1 and the second image IMG2 and disparity values corresponding to theareas. A method of merging the disparity values with the third imageIMG3 may be substantially the same as the above-described method of FIG.7 , and thus, descriptions thereof may be omitted.

According to an embodiment, the second AI model 313 may receive at leastone of the first image IMG1 and the second image IMG2. For example, thesecond AI model 313 may receive the first image IMG1, the third imageIMG3, and the disparity information Info_D, transform the third imageIMG3 based on the first image IMG1 and the disparity information Info_D,and output the transformed third image IMG3_T. As another example, thesecond AI model 313 may receive the second image IMG2, the third imageIMG3, and the disparity information Info_D, transform the third imageIMG3 based on the second image IMG2 and the disparity informationInfo_D, and output the transformed third image IMG3_T.

The first AI model 311 and the second AI model 313 may respectivelyperform neural network-based neural tasks based on various neuralnetworks. A neural network may be a model based on at least one ofvarious neural networks such as an Artificial Neural Network (ANN)model, a Multi-Layer Perceptrons (MLP) model, a Convolutional NeuralNetwork (CNN) model, a Decision Tree model, a Random Forest model, anAdaBoost model, a Multiple Regression Analysis model, a LogisticRegression model, and a RANdom SAmple Consensus (RANSAC) model. However,types of the neural network are not limited thereto. Also, a neuralnetwork for performing one task may include sub-neural networks, and thesub-neural networks may be realized as heterogeneous or homogeneousneural network models.

The first AI model 311 or the second AI model 313 may each be realizedas software, hardware, or a combination thereof. Each of the first AImodel 311 and the second AI model 313 may be trained by a manufacturerin advance and may be included in the electronic device 10 during themanufacture. However, one or more embodiments are not limited thereto,and the processor 300 may train the first AI model 311 and/or the secondAI model 313.

FIG. 11 illustrates that the image transformation module 310 includesthe first AI model 311 and the second AI model 313, but one or moreembodiments are not limited thereto. For example, the imagetransformation module 310 may include either the first AI model 311 orthe second AI model 313.

FIG. 12 is a block diagram of an electronic device 10 a according to anembodiment. In detail, FIG. 12 is a diagram of a modifiable embodimentof the electronic device 10 of FIG. 1 .

Referring to FIG. 12 , the electronic device 10 a may include a sensor100 a, a memory 200 a, and a processor 300 a. The sensor 100 a mayinclude a first image sensor 110 a and a depth sensor 120 a instead ofthe first image sensor 110 and the second image sensor 120 of FIG. 1 .The first image sensor 110 a may output a first color image C_IMG1captured in a first direction. The depth sensor 120 a may face in thefirst direction and output a depth image D_IMG2 corresponding to thefirst color image C_IMG1. The depth sensor 120 a may measure a distancevalue of an object by measuring a delay time, which is taken when pulselight from a light source is reflected from an object, and measuring adistance to the object. For example, the depth sensor 120 a may includean IR sensor.

The sensor 100 a may include a second image sensor 130 a correspondingto the third image sensor 130 of FIG. 1 . In an embodiment, the secondimage sensor 130 a may capture images in a direction perpendicular to aphotographing direction of the first image sensor 110 a and may output asecond color image C_IMG3. For example, when the first image sensor 110a photographs a front view of the host vehicle, the second image sensor120 a may photograph the side view of the host vehicle. A secondphotographing area of the second image sensor 120 a may overlap a firstphotographing area of the first image sensor 110 a.

The processor 300 a may transform the second color image C_IMG3 based onthe first color image C_IMG1 and the depth image D_IMG2 by using animage transformation module 310 a and may perform the object detectionon the second color image C_IMG3 that is transformed by using an objectdetection module 320 a.

The image transformation module 310 a according to an embodiment mayextract a target area from the first color image C_IMG1 based on thedepth image D_IMG2, instead of the disparity information of FIG. 1 .Here, the term ‘target area’ denotes an area of the first color imageC_IMG1 that may be merged with the second color image C_IMG3. In anembodiment, the image transformation module 310 a may detect an area ofthe depth image D_IMG2, which has a depth value that is equal to orgreater than a threshold value, and may extract, as the target area, anarea of the first color image C_IMG1 that corresponds to the detectedarea. In another embodiment, the image transformation module 310 a mayextract an area of the first color image C_IMG1, which overlaps thesecond photographing area of the second image sensor 130 a, as thetarget area.

The image transformation module 310 a may transform the second colorimage C_IMG3 by merging the extracted target area with the second colorimage C_IMG3.

In an embodiment, the electronic device 10 may include mappinginformation including a coordinate value of the second color imageC_IMG3 that corresponds to each coordinate value of the first colorimage C_IMG1. The image transformation module 310 a may detect thecorresponding coordinate value of the second color image C_IMG3 withregard to each pixel forming the target area, based on the mappinginformation. The image transformation module 310 a may merge a pixelvalue of each pixel forming the target area with the detected coordinatevalue of the second color image C_IMG3.

In another embodiment, for quicker calculation, the image transformationmodule 310 a may only detect a coordinate value of the second colorimage C_IMG3 that corresponds to each feature point included in thetarget area, instead of each pixel forming the target area. The imagetransformation module 310 a may merge an image having a preset size andincluding each feature point (e.g., a portion of the target area) withthe detected coordinate value of the second color image C_IMG3. Theobject detection module 320 a may perform object detection based onpixel values of the second color image C_IMG3 that is transformed.

When merging the target area with the second color image C_IMG3, theimage transformation module 310 a may merge, with the second color imageC_IMG3, the pixel values of the target area and depth values of thedepth image D_IMG2 that correspond to the target area. For example, theimage transformation module 310 a may merge the depth valuescorresponding to the target area with the second color image C_IMG3. Asanother example, the image transformation module 310 a may generatedistance information indicating a distance to the host vehicle based onthe depth values and may merge the generated distance information withthe second color image C_IMG3. A method whereby the image transformationmodule 310 a merges the depth values with the second color image C_IMG3is not limited thereto. The object detection module 320 a may performthe object detection based on the pixel values and the depth values (orthe distance information) of the second color image C_IMG3 that istransformed.

FIG. 12 illustrates that the electronic device 10 a includes the firstand second image sensors 110 a and 130 a and the depth sensor 120 a.However, according to an embodiment, the electronic device 10 a may notinclude the first and second image sensors 110 a and 130 a and the depthsensor 120 a or may include at least some of them and receive imagesfrom an external device.

FIG. 12 illustrates that the first image sensor 110 a and the secondimage sensor 130 a output color images, but according to an embodiment,the first image sensor 110 a and the second image sensor 130 a mayoutput black and white images.

FIG. 12 illustrates that the electronic device 10 a includes the memory200 a, but according to an embodiment, the electronic device 10 a may beseparated from the memory 200 a.

The electronic device 10 a according to an embodiment may detect an areaindicating a proximity object by using a color image captured in onedirection and a depth image corresponding to the color image, merge animage, which is captured in another direction and includes otherportions of the proximity object, with the detected area, and performobject detection on the merged image, thereby accurately detecting theproximity object.

Referring to FIG. 12 , the electronic device 10 a uses the depth sensor120 a, but according to an embodiment, a Light Detection and Ranging(LiDAR) sensor or a Radio Detection And Ranging (Radar) sensor may beused instead of the depth sensor 120 a.

FIG. 13 is a flowchart of an operation method of the electronic device10 a according to an embodiment. In detail, FIG. 13 is a flowchart ofthe operation method of the electronic device 10 a of FIG. 12 . At leastsome of operations of FIG. 13 may be performed by the processor 300 a ofthe electronic device 10 a.

Referring to FIGS. 12 and 13 , in operation S210, the electronic device10 a may obtain the first color image C_IMG1 captured in the firstdirection. In detail, the electronic device 10 a may obtain the firstcolor image C_IMG1 captured in the first direction by using the firstimage sensor 110 a. However, one or more embodiments are not limitedthereto, and the electronic device 10 a may obtain the first color imageC_IMG1 from the external device.

In operation S220, the electronic device 10 a may obtain the depth imageD_IMG2 captured in the first direction. In detail, the electronic device10 a may obtain the depth image D_IMG2 captured in the first directionby using the depth sensor 120 a. However, one or more embodiments arenot limited thereto, and the electronic device 10 a may obtain the depthimage D_IMG2 from the external device.

In operation S230, the electronic device 10 a may obtain the secondcolor image C_IMG3 captured in a second direction. In detail, theelectronic device 10 a may obtain the second color image C_IMG3 capturedin the second direction by using the second image sensor 130 a. However,one or more embodiments are not limited thereto, and the electronicdevice 10 a may obtain the second color image C_IMG3 from the externaldevice.

In operation S240, the electronic device 10 a may transform the secondcolor image C_IMG3 based on the first color image C_IMG1 and the depthimage D_IMG2. In detail, the electronic device 10 a may detect an areaof the first color image C_IMG1, which indicates the proximity objectclose to the electronic device 10 a, based on the depth image D_IMG2 andmay transform the second color image C_IMG3 by merging the detected areawith the second color image C_IMG3. In operation S250, the electronicdevice 10 a may perform the object detection on the second color imageC_IMG3.

FIG. 14 is a diagram of a host vehicle 400 including the electronicdevice 10, according to an embodiment. In detail, FIG. 14 is a diagramof an example of the host vehicle 400 including the electronic device ofFIG. 1 .

Referring to FIGS. 1 and 14 , the host vehicle 400 may include theelectronic device 10 and a vehicle controller 410. The electronic device10 may be disposed on the host vehicle 400, and the sensor 100 mayphotograph the front and side surfaces of the host vehicle 400. Aphotographing direction of the sensor 100 is not limited thereto, andaccording to an embodiment, the sensor 100 may photograph the rear andthe side surfaces of the host vehicle 400.

The vehicle controller 410 may control driving of the host vehicle 400overall. The vehicle controller 410 may determine situations around thehost vehicle 400 and control a navigation direction, speed, or the likeof the host vehicle 400 according to a determination result. In anembodiment, the vehicle controller 410 may receive an object detectionresult of the electronic device 10, determine the situations around thehost vehicle 400 according to the received object detection result, andtransmit a control signal to a driver (not shown) of the host vehicle400 according to a determination result, thereby controlling thenavigation direction, speed, or the like of the host vehicle 400.

Referring to FIG. 14 , the vehicle controller 410 is separated from theelectronic device 10. However, according to an embodiment, theelectronic device 10 may include the vehicle controller 410 or theprocessor 300 of the electronic device 10 and the vehicle controller 410may be integrally formed. Additionally, the electronic device mayinclude the memory 200 illustrated by FIG. 1 .

FIG. 14 illustrates that the host vehicle 400 includes the electronicdevice 10 of FIG. 1 , but according to an embodiment, the host vehicle400 may include the electronic device 10 a of FIG. 12 .

FIG. 15 is a block diagram of a self-driving device 500 according to anembodiment.

Referring to FIG. 15 , the self-driving device 500 may include a sensor510, a memory 520, a processor 530, RAM 540, a main processor 550, adriver 560, and a communication interface 570 and the above-listedcomponents of the self-driving device 500 may be interconnected to eachother via a bus. The self-driving device 500 may perform situationdetermination, navigation control, and the like by analyzing, in realtime, data of a surrounding environment of an autonomous host vehiclebased on neural networks.

The sensor 510 may include multiple sensors for generating informationregarding a surrounding environment of the self-driving device 500. Forexample, the sensor 510 may include sensors that receive image signalsregarding the surrounding environment of the self-driving device 500 andoutput the received image signals into images. The sensor 510 mayinclude an image sensor 511 such as a Charge Coupled Device (CCD) or aComplementary Metal Oxide Semiconductor (CMOS), a depth camera 513, aLiDAR sensor 515, a Radar sensor 517, and the like.

In this case, the image sensor 511 included in the sensor 510 mayinclude multiple image sensors 511. The image sensors 511 may correspondto the first image sensor 110, the second image sensor 120, and thethird image sensor 130 of FIG. 1 . Alternatively, the image sensors 511may correspond to the first image sensor 110 a and the second imagesensor 130 a of FIG. 12 . The depth camera 513 of the sensor 510 maycorrespond to the depth sensor 120 a of FIG. 12 .

The memory 520 may correspond to the memories 200 and 200 a according tothe one or more embodiments, and the processor 530 may correspond to theprocessors 300 and 300 a according to the one or more embodiments. Also,the main processor 550 may correspond to the vehicle controller 410 ofFIG. 14 . In some embodiments, the image sensor 511, the memory 520, andthe processor 530 may be realized according to the one or moreembodiments described with reference to FIGS. 1 to 15 .

The main processor 550 may control the operation of the self-drivingdevice 500 overall. For example, the main processor 550 may control afunction of the processor 530 by executing programs stored in the RAM540. The RAM 540 may temporarily store programs, data, applications, orinstructions.

The main processor 550 may control the operation of the self-drivingdevice 500 according to an operation result of the processor 530. In anembodiment, the main processor 550 may receive an object detectionresult from the processor 530 and control operation of the driver 560based on the received object detection result.

As components for driving the self-driving device 500, the driver 560may include an engine/motor 561, a steering unit 563, and a brake unit565. In an embodiment, the driver 560 may adjust acceleration, brakes,speed, directions, and the like of the self-driving device 500 by usingthe engine/motor 561, the steering unit 563, and the brake unit 565according to the control of the processor 530.

The communication interface 570 may communicate with an external devicein a wired or wireless communication manner. For example, thecommunication interface 570 may perform communication in a wiredcommunication manner such as Ethernet or in a wireless manner such asWi-Fi or Bluetooth.

As is traditional in the field, embodiments may be described andillustrated in terms of blocks which carry out a described function orfunctions. These blocks, which may be referred to herein as units ormodules or the like, are physically implemented by analog and/or digitalcircuits such as logic gates, integrated circuits, microprocessors,microcontrollers, memory circuits, passive electronic components, activeelectronic components, optical components, hardwired circuits and thelike, and may optionally be driven by firmware and/or software. Thecircuits may, for example, be embodied in one or more semiconductorchips, or on substrate supports such as printed circuit boards and thelike. The circuits constituting a block may be implemented by dedicatedhardware, or by a processor (e.g., one or more programmedmicroprocessors and associated circuitry), or by a combination ofdedicated hardware to perform some functions of the block and aprocessor to perform other functions of the block. Each block of theembodiments may be physically separated into two or more interacting anddiscrete blocks without departing from the scope of the disclosure.Likewise, the blocks of the embodiments may be physically combined intomore complex blocks without departing from the scope of the disclosure.An aspect of an embodiment may be achieved through instructions storedwithin a non-transitory storage medium and executed by a processor.

While the disclosure has been particularly shown and described withreference to embodiments thereof, it will be understood that variouschanges in form and details may be made therein without departing fromthe spirit and scope of the following claims.

What is claimed is:
 1. An electronic device comprising: a first imagesensor configured to output a first image produced by photographing afirst photographing area; a second image sensor configured to output asecond image produced by photographing a second photographing area thatoverlaps the first photographing area; a third image sensor configuredto output a third image produced by photographing a third photographingarea; and a processor configured to perform object detection on anobject included in an image, wherein the processor is configured to:generate disparity information indicating a separation degree of afeature point of the first image and the second image, extract, based onthe disparity information, a target area from the first image or thesecond image, transform the third image by merging the target area withthe third image, and perform the object detection on the transformedthird image.
 2. The electronic device of claim 1, wherein: the thirdphotographing area of the third image sensor overlaps the firstphotographing area or the second photographing area, and the processoris configured to detect an area of the first image or the second imagethat overlaps the third photographing area and extract the target areafrom the area based on the disparity information.
 3. The electronicdevice of claim 1, wherein the processor is configured to extract anarea of the first image or the second image, which has a disparity valuethat is equal to or greater than a threshold value, as the target areabased on the disparity information.
 4. The electronic device of claim 1,further comprising: a memory that stores, for each coordinate value ofthe first image or the second image, mapping information of a coordinatevalue of the third image, wherein the processor is configured to detecta first location, where the target area is merged with the third image,based on the mapping information and merge the target area with thethird image according to the first location.
 5. The electronic device ofclaim 4, wherein the processor is configured to: detect a first featurepoint included in the target area, detect a second location where thefirst feature point is merged with the third image, and merge an areahaving a preset size and comprising the first feature point with thethird image according to the second location.
 6. The electronic deviceof claim 1, wherein the processor is configured to merge, with the thirdimage, a plurality of pixel values forming the target area and aplurality of disparity values corresponding to the target area.
 7. Theelectronic device of claim 6, wherein the processor is configured toperform the object detection based on the plurality of pixel values,which form the third image that is transformed, and the plurality ofdisparity values.
 8. The electronic device of claim 1, wherein theprocessor is configured to: detect a third feature point included in thefirst image and a fourth feature point included in the second image,perform feature matching of the third feature point and the fourthfeature point, and generate the disparity information based on a resultof the feature matching.
 9. The electronic device of claim 1, furthercomprising a stereo camera comprising the first image sensor and thesecond image sensor.
 10. An electronic device comprising: a first imagesensor configured to output a first color image captured in a firstdirection; a depth sensor configured to output a depth imagecorresponding to the first color image; a second image sensor configuredto output a second color image captured in a second direction; and aprocessor configured to perform object detection on an object includedin an image, wherein the processor is configured to: extract, based onthe depth image, a target area from the first color image, transform thesecond color image by merging the target area with the second colorimage, and perform the object detection on the second color image thatis transformed.
 11. The electronic device of claim 10, wherein theprocessor is configured to extract the target area from the first colorimage, which has a depth value that is equal to or greater than athreshold value, as the target area based on the depth image.
 12. Anoperation method of an electronic device, the operation methodcomprising: obtaining a first image produced by photographing a firstphotographing area; obtaining a second image produced by photographing asecond photographing area that overlaps the first photographing area;obtaining a third image produced by photographing a third photographingarea; generating disparity information indicating a separation degree ofa feature point of the first image and the second image; extracting,based on the disparity information, a target area from the first imageor the second image; transforming the third image by merging the targetarea with the third image; and performing object detection on the thirdimage that is transformed.
 13. The operation method of claim 12,wherein: the third photographing area overlaps the first photographingarea or the second photographing area, and the extracting of the targetarea from the first image or the second image based on the disparityinformation comprises: detecting an area of the first image or thesecond image that overlaps the third photographing area; and extractingthe target area from the area that overlaps the third photographingarea.
 14. The operation method of claim 12, wherein the extracting ofthe target area comprises extracting an area of the first image or thesecond image, which has a disparity value that is equal to or greaterthan a threshold value, as the target area based on the disparityinformation.
 15. The operation method of claim 12, wherein the mergingof the target area with the third image comprises: detecting a firstlocation, where the target area is merged with the third image, based onmapping information regarding a coordinate value of the third image thatcorresponds to each coordinate value of the first image or the secondimage; and merging the target area with the third image according to thefirst location.
 16. The operation method of claim 15, wherein: thedetecting of the first location where the target area is merged with thethird image comprises: detecting a first feature point included in thetarget area; and detecting a second location where the first featurepoint is merged with the third image, based on the mapping information,and the merging of the target area with the third image according to thesecond location comprises merging, with the third image, an areacomprising the first feature point and having a preset size, accordingto the second location.
 17. The operation method of claim 12, whereinthe merging of the third image based on the disparity informationcomprises transforming the third image by merging, with the third image,a plurality of pixel values forming the target area and a plurality ofdisparity values corresponding to the target area.
 18. The operationmethod of claim 17, wherein the performing of the object detection onthe third image that is transformed comprises performing the objectdetection based on the plurality of pixel values forming the thirdimage, which is transformed, and the plurality of disparity values thatare merged.